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ABSTRACT 



The Environmental Inquiry (El) program (Cornell University 
and Pennsylvania State University) supports inquiry based, student -centered 
science teaching on selected topics in the environmental sciences. Texts to 
support high school student research are published by the National Science 
Teachers Association (NSTA) in the domains of environmental toxicology, 
watershed dynamics, biodegradation, and the ecology of invasive species. The 
first of these publications, "What's the Risk?" was published in 2001 and 
includes bioassay protocols for assessing the toxicity of substances. 
Secondary school science students can post the results of their bioassays on 
a Web server and participate in a process of anonymous peer review and 
"publication" of their research. Teachers and secondary students who have 
participated in the process reported finding it interesting and useful; 
however, we recognized that many teachers are unfamiliar with both the 
underlying science (toxicology) and the process and importance of peer review 
in scientific method. In Spring 2001, the protocol and peer review process 
was pilot tested with prospective science teachers in a secondary science 
methods course at Penn State, using a companion Web site set up specifically 
for college-level students. The results of that pilot test suggested that the 
research and peer review protocols could be adapted for use by introductory 
level college science students, including prospective science teachers. This 
paper reports the results of a multi-site expansion and test of that pilot 
work, undertaken in Fall 2001. (Author) 
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Introduction 

The Environmental Inquiry (El) program (Cornell University and Pennsylvania 
State University) supports inquiry based, student-centered science teaching on selected 
topics in the environmental sciences. Texts to support high school student research are 
being published by The National Science Teachers Association (NSTA) in the domains 
of environmental toxicology, watershed dynamics, biodegradation, and the ecology of 
invasive species. The first of these publications, What’s the Risk? , was published in 2001 
and includes bioassay protocols for assessing the toxicity of substances. Secondary 
school science students can post the results of their bioassays on a web server and 
participate in a process of anonymous peer review and “publication” of their research. 
Teachers and secondary students who have participated in the process reported finding it 
interesting and useful; however, we recognized that many teachers are unfamiliar with 
both the underlying science (toxicology) and the process and importance of peer review 
in scientific method. 

In Spring 2001 , we pilot tested the protocol and peer review process with 
prospective science teachers in a secondary science methods course at Penn State, using a 
companion website set up specifically for college-level students. The results of that pilot 
test suggested that research and peer review protocols could be adapted for use by 
introductory level college science students, including prospective science teachers. This 
paper reports the results of a multi-site expansion and test of that pilot work, undertaken 
in the Fall of 2001. 

Participants and Purpose of the Study 

This research involved college students in science courses, pre-service science 
education courses, and science studies courses at 1 1 colleges and universities around the 
United States. The overall goal of the project was to promote science education by 
engaging students in a sociologically authentic scientific research project including 
anonymous peer review. The project was designed to enable students to experience 
science as a mode of inquiry rather than a static collection of facts. 

This paper presents some preliminary quantitative data from the 1 1 -campus 
project, which was carried out in the Fall of 2001. Data are included from 10 campuses 
(the eleventh yielded only one student’s data and is omitted from the analysis). A number 
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of the faculty participants in the project will meet for the first time at the AERA 
(American Educational Research Association) and NARST (National Association for 
Research in Science Teaching) professional meetings in New Orleans in April, 2002; this 
paper is intended to be used as a resource for discussion of the project and the 
development of plans for “next steps.” 

Research Questions Guiding Initial Analysis 

1 . What do students perceive as the strengths and weaknesses of the model, rating 
the protocol specifications and written materials, the online systems, the quality of 
the reviews they received, and the extent to which they perceived that their 
experiences were scientifically "authentic?" 

2. How are the final drafts of students' research reports affected by peer reviews? 

3. Do reports improve significantly when authors receive detailed, consistent 
reviews? 

Methods of Data Collection 

All activities were organized to provide an opportunity for students to learn how 
to frame research questions, design and carry out experiments, critically analyze their 
results, write a report, and defend their conclusions to their peers. Participating students 
engaged in original research, computer-mediated collaboration, peer review, and online 
publishing. They conducted a bioassay experiment, posted their results on a web server, 
and completed anonymous peer reviews. Peer reviews were submitted using a form that 
is reproduced in Appendix B. A questionnaire with both fixed-format and open-response 
questions was administered anonymously at the end of the semester. It is reproduced in 
Appendix C. 

Students worked in pairs to conduct the bioassay experiment and tally their 
results, but posted individual reports and completed individual peer reviews. The reports 
followed a common, question-driven format, and quantitative data were entered using a 
table tool. 

After completing their own lab reports, students had about a week to complete 
online peer reviews of two other students' projects. Students composed their peer reviews 
using a structured data entry screen with two quantitative items and three essay items. 
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Peer reviews were anonymous; only report authors and instructors were given access to 
their contents. The matching of reports and reviewers was nonrandom but anonymous 
across institutions. User data, reports, and peer reviews were stored in the database in 
related tables. The final common stage of the project was "publication" of reports after 
students made revisions using peer review feedback. Since many of the major activities 
of the project occurred online (report writing, peer review, publication) most of the data 
were collected automatically. 

Data Analysis and Discussion 

We began analysis by reorganizing data tables that had been collected by our 
server using Microsoft Access. Our first task was data cleaning and the creation of one 
inclusive table by combining a user table, reports table, written reviews table, received 
reviews table, and final questionnaire table. This was a very complex process, and some 
of the methodological steps we undertook are summarized in Appendix A. Once a 
comprehensive clean data table was created in Access, we exported it to statistical 
software (SPSS) for quantitative analysis. 

There were 411 student participants. 341 (83%) gave permission for us to use 
their responses in research. We did a number of checks of participant-response bias and 
saw no meaningful differences between permission-granters and others. The following 
analyses are limited to the 341 individuals who gave consent. However, the peer review 
scores assigned to consenters by non-consenters are included, without any identifying 
information about the latter. 



2 Students at the different universities completed the experiment at different times within an approximately 
two-month time frame. Instructions to students about how to select reports to review were left to the 
instructors’ discretion. At Penn State, for example, we had our students complete the experiment first, then 
asked them to hold off on completing reviews until the results had been posted from two other institutions. 
At least one instructor encouraged his students to try to review another report that assessed the toxicity of 
the same chemical they had assessed. In most cases, however, students chose reports to review based only 
on the title of the report, which included the name of the chemical being assessed and an author-determined 
5-digit code. Lab partners shared their 5-digit codes with each other so they could avoid reviewing their 
partner’s report, which would have presented a conflict of interest. 

3 The complexity of the source data were functions of the peer review design process and the underlying 
software infrastructure. For example, most bioassay reports were reviewed by two different reviewers: in 
database terms, this is a “one-to-many relationship” between report and reviews, and such data are typically 
stored in separate tables. Furthermore, each author wrote two reviews, and these reviews provide 
information related to the review author as well as the source report, so there is a second one-to-many 
relationship between users and reviews. 
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In the following pages, data are presented as were gathered by the automated 
system. We provide discussion to address related issues and their relevance where 
necessary. 



Are you in a teacher education program? 



Although there were teacher education students at 




Frequency 


Percent 


most of the participating colleges, they were outnumbered by 


No 


203 


68.4 


science majors. 44 participants’ major could not identified 


Yes 


94 


31.6 


(this information was provided in the final questionnaire, 


Total 


297 


100.0 



which not all consenters completed); therefore out of 341 

participants, 297 are reported in this table. The following table reports the number of students and 
whether they are in a teacher education program, by school. 



University 


| School 
code 


i 


3 


4 


5 j 


6 


7 


8 


10 


11 ; 


12 


Total 


Number of students 


Not a 
teacher ed 
student 




20 






1 


123 




1 


9 


49 


203 


Teacher ed 
student 


16 




28 


12 


2 


5 


11 




20 




94 


Total 
teacher ed 
status 
known 


16 


20 


28 1 


12 


3 j 


128 | 


11 


1 


29 ? 


49 


297 


Teacher 

ed status 0 000 3 0 0 9 31 0 43 

unknown 



Missing values = 44, 12.9% of the total N of 341 consenters. 

One non-consenting participant is omitted, the only student from an 1 1 th university. 
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What are your gender and minority group affiliations? 

74 participants (21.7%) were male. Statistical analyses 
did not yield significant differences on any variables between 
male and female students. Differences among schools in 
gender distribution were not statistically significant. With the 
exception of one school, universities with more than six 
participants all had female participants outnumbering male participants by at least three to one. 
This was true among science courses as well as science education courses. 

17.6% of the students who completed the final questionnaire identified themselves as 
members of underrepresented minority groups (African-American, Hispanic, Native American). 
There were no statistically significant differences associated with this response on any measure. 
6.2% reported that English was not their first language; no statistically significant differences 
were detected for this measure either. 

Basic descriptive statistics for the final student questionnaire 

Of the 341 students who submitted reports and gave consent for research, 192 
(57% of consenters) completed the final questionnaire. Summary statistics from the 
questionnaire (reproduced in Appendix C) are reported below. We used Likert-scale 
items, where 3 = “Neutral,” 4 = “Agree somewhat,” and 5 = “Strongly agree.” 





Frequency 


Percent 


Female 


264 


78.1 


Male 


74 


21.9 


Total 


338 


100.0 





Descriptive Statistics 


N 


Mean for 

all 

students 


Mean for 
teacher ed 
students 


Mean for 
other 
students 


1 


I learned something by writing peer review comments 


192 


3.96 


3.82 


4.05 


2 


1 felt qualified to provide meaningful peer review of other 
students' reports 


192 


3.73 


3.65 


3.78 


3 


1 believe that the peer reviews I wrote should be helpful to the 
students that received them 


192 


3.98 


3.97 


3.99 


4 


Peer reviewing other students has helped me to think more 
critically 


193 


4.10 


4.08 


4.11 


5 


Peer reviewing other students has helped me to improve my 
own scientific writing 


193 


4.02 


3.90 


4.08 


6 


1 received useful peer review comments about my own report 


192 


3.53 


3.36 


3.63 


7 


The quantitative scores I received from peer reviewers were 
fair 


192 


3.60 


3.51 


3.66 


8 


1 changed my mind about something in my report because of 
comments I received through peer review 


192 


2.99 


2.94 


3.02 
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9 


It is easier to say what I really think when 1 don't have to sign 
my name or meet in person with the students 


192 


3.71 


3.69 


3.72 


10 


I think that meaningful peer review is a reasonable expectation 
for college students 


190 


4.23 


4.21 


4.24 


11 


I think that meaningful peer review would be a reasonable 
expectation for high school students 


190 


3.88 


3.96 


3.84 



None of the above differences is statistically significant at p < .05. 

Although teacher education student means were lower for all items except item 1 1, these 
differences are not statistically significantly (ANOVA with correction for multiple t-tests). 
However, it is worth noting that item 1 1 evaluates high school students’ ability to provide sound 
feedback to each other. 

We provide individual descriptive statistics and bar graphs for each final questionnaire 
item below. 

“I learned something by writing peer review comments.” 






Freq. 


% 


Strongly disagree 


2 


1.0 


Disagree 


7 


3.6 


Neutral 


32 


16.7. 


Agree 


106 


55.2 


Strongly agree 


45 


23.4 


Total 


192 


100 



disagree agree 

A majority of the respondents (79%) agreed that they learned something by writing peer review 
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“I felt qualified to provide meaningful peer review of other students 1 reports.” 






Freq. 


% 


Strongly disagree 


7 


3.6 


Disagree 


18 


9.4 


Neutral 


34 


17.7 


Agree 


94 


49.0 


Strongly agree 


39 


20.3 


Total 


192 


100 



disagree agree 

79% of the students reported that they felt qualified to provide meaningful reviews of other 
students’ reports. . 



M I believe that the peer reviews I wrote should be helpful to the students that received 
them.” 





Freq. 


% 




Strongly disagree 


3 


1.6 


50 


Disagree 


2 


1.0 


40 


Neutral 


29 


15.1 


30 


Agree 


119 


62.0 




Strongly agree 


39 


20.3 


20 


Total 


192 


100 


c 10 



O) 

a 




strongly disagree 



neutral 



strongly agree 



disagree 



agree 



82% of the students thought they provided helpful reviews, and less than 3% anticipated that their 
review would not be helpful. 
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“Peer reviewing other students has helped me to think more critically.” 






Freq. 


% 


Strongly disagree 


6 


3.1 


Disagree 


1 


.5 


Neutral 


29 


15.0 


Agree 


89 


46.1 


Strongly agree 


68 


35.2 


Total 


193 


100 



disagree agree 

82% of the students agreed that peer reviewing enabled them to reflect and think about their own 
and others’ research more critically. 



“Peer reviewing other students has helped me to improve my own scientific writing.” 

50 





Freq. 


% 


40 


Strongly disagree 


2 


1.0 




Disagree 


8 


4.1 


30 


Neutral 


37 


19.2 




Agree 


84 


43.5 


20 


Strongly agree 


62 


32.1 




Total 


193 


100 


10 



c 



strongly disagree neutral strongly agree 

disagree agree 



Providing feedback on other students’ research reports was perceived beneficial by students. 75% 
of the respondents agreed that their technical writing improved because of the peer reviewing 
process. 



a 
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“I received useful peer review comments about my own report.” 






Freq. 


% 


Strongly disagree 


10 


5.2 


Disagree 


22 


11.5 


Neutral 


51 


26.6 


Agree 


74 


38.5 


Strongly agree 


35 


18.2 


Total 


192 


100 



disagree agree 

Although 82% of the students thought they provided helpful reviews, only 57% reported that they 
received helpful reviews. 18% of students reported that peer reviews did not help them to 
improve their reports. 

“The quantitative scores I received from peer reviewers were fair.” 






Freq. 


% 


Strongly disagree 


7 


3.6 


Disagree 


14 


7.3 


Neutral 


69 


35.9 


Agree 


60 


31.3 


Strongly agree 


42 


21.9 


Total 


192 


100 



- disagree agree 

Most of the students thought their peers were fair when they rated the quality of the reports. 
Previous research has shown that marks given by students can be as reliable as those given by 
instructors (Orpen, 1982). 1 1% of the participants reported that their score were “unfair.” 
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“I changed my mind about something in my report because of comments I received through 
peer review.” 






Freq. 


% 


Strongly disagree 


30 


15.6 


Disagree 


38 


19.8 


Neutral 


49 


25.5 


Agree 


54 


28.1 


Strongly agree 


21 


10.9 


Total 


192 


100 



disagree agree 

39% of the students agreed that they changed their minds about some aspect of their report 
because of feedback they received via peer review. This might be attributed in part to the 
implications of peer evaluation, which involve a different relationship that that between 
instructors and students. It may contribute to a collaborative role rather than an adversarial one 
(Billington, 1997). 

“It is easier to say what I really think when I don f t have to sign my name or meet in person 
with the students.” 






Freq. 


% 


Strongly disagree 


13 


6.8 


Disagree 


20 


10.4 


Neutral 


36 


18.8 


Agree 


63 


32.8 


Strongly agree 


60 


31.3 


Total 


192 


100 



disagree agree 
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A majority of students felt positive about the anonymity of peer review. This is consistent with 
what actually happens in scientific community. According to Arnold Reiman, the chief editor of 
the New England Journal of Medicine, about 85% of their reviewers have preferred to remain 
anonymous, and report that they are more candid and rigorous when they are not required to sign 
their reviews. 

“I think that meaningful peer review is a reasonable expectation for college students.” 






Freq. 


% 


Strongly disagree 


4 


2.1 


Disagree 


2 


1.1 


Neutral 


17 


8.9 


Agree 


91 


47.9 


Strongly agree 


76 


40.0 


Total 


190 


100 



disagree agree 
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“I think that meaningful peer review would be a reasonable expectation for high school 
students.” 






Freq. 


% 


Strongly disagree 


5 


2.6 


Disagree 


9 


4.7 


Neutral 


33 


17.2 


Agree 


87 


45.3 


Strongly agree 


58 


30.2 


Total 


192 


100 



disagree agree 



Students were consistently positive about their experiences writing and receiving peer reviews. 
87% thought college students could provide meaningful and helpful peer reviews. Previous 
research has suggested that students appreciate the opportunity to comment on each other’s work 
in a constructive manner, and that peer review can instill a sense of community within a class 
(Hay & Miller, 1992). When students were asked if it was realistic to expect meaningful reviews 
from high school students, 75% responded positively. There is no significant difference between 
teacher education students and other students on this measure. However as noted earlier, this item 
was the sole item on which teacher education students felt more positive than other students. 

School Differences in Quantitative Review Scores 

In their peer reviews, students rated the quality of the argument and the quality of authors’ 
technical writing by assigning a score to each. We found some statistically significant differences 
between schools. An ANOVA procedure was used to detect these differences and then post hoc 
analyses were done to identify pairwise differences between schools. 

The first measure, which we call QScorel, asked reviewers to answer the question, “Did 
the author address each question fully and provide good support for his or her conclusions?” 
Responses were reported on a five-point scale ranging from 5 = “Excellent. Exceptionally well 
done” to 1 = “Failure. Unacceptable responses; report should be restarted from scratch.” We call 
this the “quality of argument” score. Students at School 6 received significantly higher scores on 
this measure than students at Schools 3,10, and 12. Because School 6 had a small number of 
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participants (n=6), this result should be carefully interpreted. There were no other pairwise 
differences. 

Score 1 (Quality of Argument) 

One-Way Analysis of Variance 
Dependent Variable: QSCOIRAV (Avg QScorel RECEIVED) 



Source 


DF 


Sum of Squares 


Mean Square 


F Value 


Pr > F 


SCHOOL 


9 


11.8739413 


1.3193268 


2.53 


0.0082 


Error 


309 


161.0752053 


0.5212790 






Corrected Total 


318 


172.9491466 









Post hoc tests 



Duncan Grouping 


Mean 


N 


SCHOOL 


A 


3.8000 


5 


6 


B 


2.6973 


49 


12 


B 


2.5877 


19 


3 


B 


2.5000 


, 9 


10 



Significant differences at p<.05. Means with the same letter are not significantly different 

ANOVA also revealed a significant school effect on QScore Is written; however, we are 
still trying to interpret this finding, because we do not see any obvious differences in mean scores. 



One-Way Analysis of Variance 
Dependent Variable: QSCOIWAV (Avg QScorel WRITTEN) 



Source 


DF 


Sum of Squares 


Mean Square 


F Value 


Pr > F 


SCHOOL 


. 10 


13.3476149 


1.3347615 


3.19 


0.0006 


Error 


366 


153.1733886 


0.4185065 






Corrected Total 


376 


166,5210035 
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Score 2 (Quality of Technical Writing) 

There were significant differences among schools in scores received for quality of 
technical writing (QScore2). One-way ANOVA was performed, followed up with Duncan 
grouping post hoc analysis for pairwise comparisons. Three groups of schools were identified, as 
seen in the table below, with statistically different average received mean scores. Schools 6 and 5 
comprised two discrete “groups,” A and B. Schools 1, 3, 7, and 12 comprise a third group with a 
significantly different mean score, when compared to Groups A & B. There were no other 
differences. 



One-Way Analysis of Variance 
Dependent Variable: QSC02RAV Avg Qscore2 RECEIVED 



Source 


DF 


Sum of Squares 


Mean Square 


F Value 


Pr > F 


SCHOOL 


9 


14.9449277 


1.6605475 


3.53 


0.0003 


Error 


309 


145.3587557 


0.4704167 






Corrected Total 


318 


160.3036834 









Post hoc tests 



Duncan Grouping 


Mean 


N 


SCHOOL 


A ' 


3.9000 


5 


6 


B 


3.2083 


12 


5 


C 


2.5986 


49 


12 


C 


2.5767 


126 


7 


c 


2.5313 


16 


1 


c 


2.4912 


19 


3 



Significant differences at p<.05. Means with the same letter are not significantly different 

The following ANOVA table shows that students at School 6 awarded significantly 
higher scores to others concerning the technical quality of reviewed reports, an interesting 
phenomenon given that they also received the highest scores. Students School 5 awarded 
significantly lower scores; however, they received the second highest scores for their reports. 
(Please note that these are only preliminary analyses; we still need to look at issues like which 
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schools tended to review which other schools. Again, the matching of reports to reviewers was 
anonymous but not random, and it is likely that students were most likely to review reports by 
other students from their own campus, because their reports were most likely to be available for 
review at the time each campus’s reviews were required by the relevant instructor.) 



One-Way Analysis of Variance 
Dependent Variable: QSC02WAV (Avg QScore2 WRITTEN) 



Source 


DF 


Sum of Squares 


Mean Square 


F Value 


Pr > F 


SCHOOL 


10 


13.8479459 


1.3847946 


2.91 


0.0017 


Error 


298 


141.8592512 


0.4760378 






Corrected Total 


308 


155.7071971 









Post hoc tests 



Duncan Grouping 


Mean 


N 


SCHOOL 


A 


3.5000 


4 


6 


B 


2.5656 


122 


7 


B 


2.4333 


5 


10 


B 


2.2222 


12 


5 



Significant differences at p<.05. Means with the same letter are not significantly different 



Differences in Quantitative Review Scores for Teacher Education Students 

Students in teacher education programs generally received and assigned higher 
mean scores than non-teacher education students. However, among the differences in 
mean scores for all four measures, the only statistically significant difference concerned 
the average score received for the quality of technical writing. Teacher education students 
were able to articulate their research and communicate results in a more effective way 
than the students who are majored in sciences or science studies. 
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Written and Received Score Differences in Reviews for Teacher Education Majors 



Teacher 


Avg QScore 1 


Avg QScore2 


Avg QScorel 


Avg QScore2 


Education 


RECEIVED 


RECEIVED* 


WRITTEN 


WRITTEN 


No 


2.7508 


2.60017 


2.7362 


2.6503 


Yes 


2.9147 


2.84425 


2.8653 


2.7991 



*Only the received quality of technical writing score (QScore2) is statistically significant at p<.05. 



QScore 2 (Quality of Technical Writing) Received 

One-Way Analysis of Variance 
Dependent Variable: QSC02RAV Avg QScore2 RECEIVED 



Source 


DF 


Sum of Squares 


Mean Square 


F Value 


Pr > F 


TCHRED 


1 


3.5135934 


3.5135934 


7.08 


0.0082 


Error 


280 


138.9269355 


0.4961676 






Corrected Total 


281 


142.4405290 









Final Comments 

As noted at the beginning of this paper, these are preliminary findings from this 
project, intended as background information to stimulate subsequent discussion and 
analysis by participating faculty and other interested researchers. 

In looking for differences by school and other factors, our primary interest was in 
developing questions to guide formative evaluation of this project. For example, what are 
the advantages and disadvantages of restricting participation in a project like this to 
prospective science teachers? Do between-school differences lead to differences in 
review-related outcomes? Do positive experiences as a reviewer and as a review-receiver 
favorably incline preservice teacher participants to consider using peer review with their 
own students some day? 

There are also a number of questions that we have not yet begun to analyze 
concerning the validity and reliability of evaluations by students in this project. We invite 
the participation of our CPR collaborators in undertaking these next steps. 
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Appendix A: Methodological Notes on Data Analysis 

In MS Access: 

1 . We removed duplicate records from the Final Questionnaire using the following 
rules: (a) Delete incomplete entries, (b) Delete older entries when multiple 
complete entries exist. 

2. We assigned dummy user names to anonymous entries (anonl, anon2, etc.). 

3. We ran a “find duplicates query” on Peer Reviewed Report and found many 
unsubmitted duplicates. These were all deleted. 

4. We had nine students who double-posted their reports and got reviews for both 
versions. We pooled all reviews for these nine people by deleting one copy each 
and pointing the original reviews to the remaining duplicate. 

5. We also deleted duplicate peer reviews only if it would be clear to the report 
author that a duplicate had been submitted. There were three pairs of duplicates 
where the wording and/or scores were different and we judged that an author 
might have concluded that they came from different reviewers. 

6. There were a number of duplicate records in the User table identified by running 
a “find duplicates query” on password. 

7. There were 23 pairs of reports with identical titles. Only one pair was completely 
identical, submitted by UIDs 620 and 629. 

8. There were four users who did not submit reports but submitted peer reviews. 
There were 34 users who submitted reports but not peer reviews. 

9. Seven reports that were not submitted are removed to the PRReport deleted table. 

10. Before removing non-consenters there were 41 1 reports. We omitted people who 
did not give their consent to use their data for research, resulting in 341 
participants. 




J 
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Appendix B: Web-Based Form Used for Student Peer Reviews 
(The following content comprises the web-based form used by students to complete peer 
reviews). 

1 . Did the author address each question fully and provide good support for his or her 
conclusions? 

• Excellent. (Exceptionally well done). 

• Good. (Very well done). 

• Satisfactory. (Acceptable response to all questions). 

• Poor. (Minimal attention to the questions and/or serious technical problems with 
one or more responses). 

• Failure. Unacceptable responses; report should be restarted from scratch. 

2. Please rate the overall quality of the writing in this report for clarity, readability, and 
technical accuracy (spelling errors, run-on sentences, etc.). 

• Excellent. (Exceptionally well done). 

• Good. (Very well done). 

• Satisfactory. (Acceptable response to all questions). 

• Poor. (Minimal attention to the questions and/or serious technical problems 
with one or more responses). 

• Failure. (Unacceptable responses; report should be restarted from scratch). 

3. What was a particular strength in this experimental design? (Scrolling text box) 

4. Do you agree with the conclusions? Do they appear to be supported by the results of 
the experiment? (Scrolling text box) 

5. What suggestions can you make for improving the experimental design? (Scrolling text 
box) 
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Appendix C: Final Questionnaire 

Final Questionnaire: College Peer Review Project 

You were asked earlier to help us evaluate the College Peer Review project by 
completing a questionnaire about your experience. If you are willing to complete this 
optional step, thank you! Evaluation will help us determine the value of the project and 
will guide the project's future development. If you would like to see the original consent 
form in a new window, click here . Otherwise, simply fill in the information below. 



T op of Form 



.Demographic Questions -V . ■ * * 

(Who participated) . ; ' : f ' c ir" : ' ~:t " 


Please note that if you wish to not responded aiiy question,^ 
leave it blank. J ' t . Vj/*, f.iv .! !•!*«£ : •*'*. - ' r-V*. 


Your User ID (e.g., jas2 1 ) 


...... J 


Your 4-digit password 


i 


Your student status 


r r r 

Undergraduate Graduate Other (e.g., non-degree) 


What is your current major field of 
study? 


j 


| If you have a dual major, what is your 
second major field of study? 


1 _l ' ••• . 


Are you in a teacher education program? 


r r 

Yes No 


When do you plan to complete your 
current degree? 


1 2001 S 


What is your gender? 


r c 

Male Female 


Do you consider yourself a member of 
one or more of the following minority 
groups: African-American, Hispanic, 

1 Native American? 


or 

Yes No 


Was English your first language? 


r n 

Yes No 


Have you ever done scientific research in 
a setting other than a conventional college 
class? 


re 

Yes No 


If you answered "yes" to the previous 
question, please describe the research, the 
setting, and the length of time you were 
involved 


jg 

‘ ® 

gj 



Please continue below: 
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I 


, Please make one choice in each • . 

, . : . , >' .row : • 


|process/Outcome Questions (5-point scale) . 


Strongly 

disagree 


Disagree j 
somewhat 


Neutral 


’ Agree 
somewhat ' 


Strongly’. 

agree 


I learned something by writing peer review comments. 


c 


c 


r 


r 


c 


j I felt qualified to provide meaningful peer review of other 
students' reports. 


r 


r 


r 


c 


c 


1 1 believe that the peer reviews I wrote should be helpful to 
the students that received them. 


C \ 


r 


r 


c 


o 


Peer reviewing other students has helped me to think more 
critically. 


r 


r 


c 1 


o 


0 


Peer reviewing other students has helped me to improve my 
own scientific writing. 


c ! 


r \ 


c 


r 1 


c 


I received useful peer review comments about my own 
report. 


c 


r 


c 1 


r 


c 


|The quantitative scores I received from peer reviewers were 
fair. 


c 


c 


c 


c 


c 



Continuation of above table; i 


,Strongly ‘ 
.disagree » 


* Disagree 
somewhat • 


!' Neutral 


Agree . 
somewhat 


Strongly 

agree: 


I changed my mind about something in my report because of s 
comments I received through peer review. 


o 


G 


c 


G 


c 


It is easier to say what I really think when I don't have to 
sign my name or meet in person with the students who wrote 
the research reports. 


c 


c 


c i 


C 


o 


I think that meaningful peer review is a reasonable 
expectation for college students. 


r 


c 


r | 


r 


o 


I think that meaningful peer review would be a reasonable 
expectation for high school students. 


c 


r 


c ! 


o 


c 


If 1 taught science, I would like to use some type of formal i 
student peer review. 


r 


r 


r 


o , . 


c 
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Please continue: 




When you finish, press the "Submit Questionnaire" button. 



Subnit Questionnaire.. 



Bottom of Form 
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